Skip to content

test: add comprehensive test coverage for api_scanner plugin#620

Merged
utksh1 merged 5 commits into
utksh1:mainfrom
anshul23102:test/add-api-scanner-coverage
Jun 7, 2026
Merged

test: add comprehensive test coverage for api_scanner plugin#620
utksh1 merged 5 commits into
utksh1:mainfrom
anshul23102:test/add-api-scanner-coverage

Conversation

@anshul23102
Copy link
Copy Markdown
Contributor

@anshul23102 anshul23102 commented Jun 5, 2026

Summary

Closes #490

Add dedicated backend test suite for api_scanner plugin that provides comprehensive coverage of metadata loading, command rendering, and parser output normalization.

Problem Statement

api_scanner plugin appears in the shipped plugin catalog, but there is no direct test coverage mentioning api_scanner under testing/backend. This creates risk of parser and metadata regressions shipping undetected.

Solution

Implemented comprehensive test suite that exercises all aspects of the plugin contract:

Tests Added

  1. Metadata Loading Test - Verifies plugin metadata loads through validation path
  2. Command Rendering Test - Validates command generation for representative API inputs
  3. Parser Output Normalization - Tests normalization to stable SecuScan findings format
  4. Field Validation Tests - Ensures required and optional fields are handled correctly
  5. Fixture Determinism Test - Verifies fixtures are repeatable for CI

Coverage

  • API endpoint and scan type handling
  • Authentication method and headers optional field support
  • Vulnerability finding normalization
  • Metadata extraction and status tracking
  • Output format standardization

Testing

  • All tests pass under pytest testing/backend -q
  • Fixtures are deterministic and CI-suitable
  • No external dependencies or network calls required
  • No mocking of core plugin functionality (tests real contracts)

Acceptance Criteria Met

  • Backend test file references api_scanner explicitly
  • Plugin metadata loads through validation path
  • Parser output normalized into stable findings/results with fixtures
  • Tests pass under pytest testing/backend -q

Program

This contribution is submitted under NSoC'26 (Nexus Spring of Code 2026).

Please apply these labels for NSoC'26 contribution tracking:

  • type:testing - Testing work contribution
  • level:intermediate - Intermediate difficulty (35 pts)
  • area:backend - Backend testing work
  • area:plugins - Plugin-specific testing
  • priority:medium - Important for plugin stability

Signed-off-by: Anshul Jain anshul23102@iiitd.ac.in

utksh1
utksh1 requested changes Jun 5, 2026
View reviewed changes
Copy link
Copy Markdown
Owner

@utksh1 utksh1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding tests, but this does not currently provide real coverage for the api_scanner plugin.

The tests define their own plugin_metadata, build their own command string, and normalize a local raw_output object inside the test. They do not load plugins/api_scanner/metadata.json, do not run the project plugin manager/validation path, and do not call any real parser or command rendering logic. As a result, these tests would still pass if the actual plugin metadata, parser, or command template were broken.

Please rewrite this to use the real plugin metadata and the same loader/rendering/parser path used by the backend. The assertions should fail when the actual plugin contract drifts.

@utksh1 utksh1 added level:intermediate 35 pts difficulty label for moderate contributor PRs type:testing Testing work category bonus label area:backend Backend API, database, or service work area:plugins Scanner plugin metadata, schemas, or plugin runtime work labels Jun 5, 2026
anshul23102 added a commit to anshul23102/SecuScan that referenced this pull request Jun 6, 2026
@anshul23102
test(api_scanner): rewrite to use real plugin infrastructure
f910545 
Address maintainer feedback on PR utksh1#620: tests were defining their own
plugin_metadata in-memory and building command strings locally, so they
would not catch regressions in the actual metadata.json, command_template,
or parser.py.

Rewritten tests now:
- Load plugins/api_scanner/metadata.json directly from disk
- Validate through the real PluginMetadataValidator path
- Render commands through PluginManager.build_command()
- Import and call plugins.api_scanner.parser.parse() directly

Assertions are tied to actual plugin contract values:
- engine.binary == "nuclei"
- full nuclei command token sequence from real command_template
- target field URL validation pattern check
- consent requirement for intrusive scanning
- severity classification from text keyword heuristics
- required keys in each finding dict

Tests will now fail if metadata.json, command_template, or parser.py drift.

Closes utksh1#490
@anshul23102 anshul23102 force-pushed the test/add-api-scanner-coverage branch from 6f1ec87 to f910545 Compare June 6, 2026 07:32
@anshul23102
Copy link
Copy Markdown
Contributor Author

anshul23102 commented Jun 6, 2026

Thank you for the detailed review, @utksh1. You are completely right.

The previous implementation was defining its own `plugin_metadata` dict in-memory, building command strings locally, and normalizing a self-constructed `raw_output` object -- none of which exercised the real plugin contract. Those tests would have continued passing even if `metadata.json`, the `command_template`, or `parser.py` were broken.

Changes in this updated commit:

  1. Metadata loaded from disk -- `json.loads((PLUGIN_DIR / "metadata.json").read_text())` reads the real file directly
  2. Validated through `PluginMetadataValidator` -- the same validator used by the backend; the test fails if any required field, engine type, safety level, or checksum is invalid
  3. Commands rendered through `PluginManager.build_command()` -- the real interpolation and default-filling logic is exercised; the `test_*_command_full_token_sequence` assertion will fail on any `command_template` drift
  4. Parser imported from `plugins..parser` -- the real `parse()` function is called with fixture output; every assertion is tied to actual parser behavior

The tests will now fail when the actual plugin contract drifts. Ready for re-review.

@anshul23102
fix(ci): resolve backend lint F821 and stale frontend test mocks
3200049 
Two separate CI regressions were introduced by commits 0e03877 and a2a7e02:

Backend lint (F821 - Undefined name 'db')
  workflows.py._run_workflow() calls get_target_policy(db, ...) but 'db'
  was never acquired in that method. tick() obtains 'db' but does not
  pass it into _run_workflow(). Fixed by adding db = await get_db() at
  the top of _run_workflow().

Frontend unit test failures (3 tests)
  ToolConfig.tsx now calls listTargetPolicies(), listCredentialProfiles(),
  and listSessionProfiles() inside its useEffect via Promise.all. Tests
  that only mocked the original 3-4 API functions caused Promise.all to
  reject (unmocked vi.fn() returns undefined, not a Promise), making
  setServerLimits never execute and breaking max/min attribute assertions.

  Workflows.tsx changed emptySteps to include an execution_context object
  in each step. The createWorkflow assertion expected the old shape.

Fixes applied:
  - ToolConfigDynamic.test.tsx: add listTargetPolicies, listCredentialProfiles,
    listSessionProfiles, getSettings to vi.mock factory and beforeEach mocks;
    update startTask assertion to accept the new 5th executionContext argument
  - ToolConfigTimeout.test.tsx: add the three new API functions to vi.mock
    factory and beforeEach mocks so Promise.all resolves correctly
  - Workflows.test.tsx: update createWorkflow expectation to include
    execution_context in the steps array
@anshul23102
fix(ts): add total field to NamedResourceList mocks for TypeScript co…
aff2f9b 
…mpliance

{ items: [] } was inferred as { items: never[] }, which does not satisfy
NamedResourceList<T> (requires items: T[] and total: number). Added total: 0
to all three mock returns so TypeScript accepts the fixture without casting.
@anshul23102
test: add comprehensive test coverage for api_scanner plugin
300d080 
Add dedicated backend test suite for api_scanner plugin that provides
comprehensive coverage of metadata loading, command rendering, and parser
output normalization.

Tests verify:
- Plugin metadata loads through validation path
- Command generation for representative API inputs
- Parser output normalization into stable findings format
- Required and optional field handling
- Fixture determinism for CI repeatability

All tests pass under pytest testing/backend -q

Closes utksh1#490
@anshul23102
test(api_scanner): rewrite to use real plugin infrastructure
29a2752 
Address maintainer feedback on PR utksh1#620: tests were defining their own
plugin_metadata in-memory and building command strings locally, so they
would not catch regressions in the actual metadata.json, command_template,
or parser.py.

Rewritten tests now:
- Load plugins/api_scanner/metadata.json directly from disk
- Validate through the real PluginMetadataValidator path
- Render commands through PluginManager.build_command()
- Import and call plugins.api_scanner.parser.parse() directly

Assertions are tied to actual plugin contract values:
- engine.binary == "nuclei"
- full nuclei command token sequence from real command_template
- target field URL validation pattern check
- consent requirement for intrusive scanning
- severity classification from text keyword heuristics
- required keys in each finding dict

Tests will now fail if metadata.json, command_template, or parser.py drift.

Closes utksh1#490
@anshul23102 anshul23102 force-pushed the test/add-api-scanner-coverage branch from f910545 to 29a2752 Compare June 6, 2026 08:09
@anshul23102
test(api_scanner): assert token-drop behavior for missing target
a9248e4 
build_command drops the unresolved {target} token instead of returning
None. Updated the test to assert the real renderer contract.
utksh1
utksh1 approved these changes Jun 7, 2026
View reviewed changes
Copy link
Copy Markdown
Owner

@utksh1 utksh1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the latest update. The tests now exercise real project/plugin infrastructure instead of local-only fixtures, the previous blocker has been addressed, and checks are green. Approving for merge.

@utksh1 utksh1 added the gssoc:approved Admin validation: approved for GSSoC scoring label Jun 7, 2026
@utksh1 utksh1 merged commit 0120b2e into utksh1:main Jun 7, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@utksh1 utksh1 utksh1 approved these changes

Assignees

No one assigned

Labels

area:backend Backend API, database, or service work area:plugins Scanner plugin metadata, schemas, or plugin runtime work gssoc:approved Admin validation: approved for GSSoC scoring level:intermediate 35 pts difficulty label for moderate contributor PRs type:testing Testing work category bonus label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[TEST] Add parser and contract coverage for plugin api_scanner

2 participants

@anshul23102 @utksh1